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Abstract- Winner-take-all (WTA) neural networks using pulse-firing process- 
ing elements are introduced. In the pulse-firing W"TA (PWTA) networks de- 
scribed, input and activation signal shunting is controlled by one shared lat- 
eral inhibition signal. This organization yields an O(n) area complexity that is 
convenient for integrated circuit implementation. Appropriately specified net- 
work parameters allow for the accurate continuous evaluation of inputs using a 
signal representation compatible with established pulse-firing neural network 
implementations. 

1 Introduction 

The winner-take-all (WTA) function plays a central role in competitive neural networks 
and is related to recurrent on-center off-surround models of natural neural systems [1-3]. 
Although it can be realized sequentially via pairwise comparisons, the WTA operation is 
more effectively realized in parallel analog circuits via a distributed network of processing 
elements which compare relative input magnitudes and allow only that element with the 
largest input (or ” winner”) to remain active. Parallel analog WTA realizations have been 
described which use Hopfield Network dynamics [4], and MOS current conveyors [5,6]. 
The model introduced in this paper and its electronic implementation are more like a 
WTA mechanism inspired by natural presynaptic inhibition feedback [7]. The new pulse- 
firing WTA (PWTA) model employs a unique combination of a self-shunting feedback 
term with output hysterisis to yield a WTA network compatible with asynchronous pulse- 
firing neural network implementations described variously as impulse, pulse-stream, and 
neural-type networks [8-10]. 

This paper first introduces asynchronous pulse firing processing units in Section 2. 
These are the basic computational units used in PWTA networks. The mathematical 
foundation of PWTA networks is then presented in Section 3 where the system dynamics 
of a general PWTA network are developed. Section 4 continues with the presentation of 
MOS circuit implementations. Section 5 closes with an analysis of finite circuit precision 
effects. 


2 Asynchronous Pulse Firing Processing Units 

The dynamics of the pulse firing processing units used in a PWTA network obey the 
following equations: 


5 . 1.2 


kv — -av + x - (fiv + x - y )g(v), v(t 0 ) = 0 ( 1 ) 

y = g(v) 

where v is unit activation, i is total unit input, y is unit output, and g(-) is the binary 
hysterisis function shown in Figure 1. As can be deduced from the figure g(v) includes as a 
special case the simple threshold nonlinearity (when V t i = V t h ) although the specific pulse 
firing dynamics described here would cease to exist in that situation. Throughout this 
paper input x is assumed to be positive and time variant. The unit response to a constant 
input is a train of regularly spaced constant width pulses. The larger the input signal 
i, the greater the output pulse repetition rate. The parameter a establishes a first-order 
response to x during the input integration phase of operation as defined by the absence of 
an output pulse (p — 0). That response is shifted to one defined by a -{■ f 3 during the firing 
phase, as defined by the presence of an output pulse ( y = 1). Since x is shunted during the 
output pulse period, processing element state asymptotically approaches e = 7 /(a + /?). 
Parameter « uniformly scales all unit time constants. One pulse firing cycle is summarized 
by the integration of the input signal until v reaches V t h, whereupon the switches toggle, 
causing the discharge of v to min(e, V t i). Oscillation is sustained provided e < Vjj. 
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Figure 1: Output hysterisis function 

3 PWTA Network Dynamics 

A PWTA network combines the unit dynamics described in (2) with lateral inhibition. Lat- 
eral inhibition from a combination of unit outputs can be expressed in a form which yields 
network state equations similar to those of the presynaptic inhibition model described by 
Yuille and Grzywacz [7]: — i. •• • r 

KVi = -avi + XiF{V) - H(V) ( 2 ) 
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where 


F(V) = 1-Vff(v k ) 

k 

and 

ff(V) = ((A - /3)v< + 7 - <)s(oi) + {fa - 7 )V<K”0 

k 

with V indicating the logical OR operation and V corresponding to the vector of unit 
activations. F(V) is a binary value establishing the input inhibition which occurs while 
any processing element generates a pulse. It is during this ’’output firing phase” of the 
network that H(V) controls the processing elements such that a winning unit activation 
decays at a different rate to a different equilibrium than that of a losing unit. The model 
parameters allow for the independent adjustment of winning and losing unit decay rates 
and asymptotes. Properly chosen parameter values guarantee WTA function independent 
of initial system state without the need for external synchronization [11]. 

All units contribute to a shared lateral inhibition signal identically. The winning output 
is indicated by the dominance of the first unit activation to reach V t h- since it establishes 
the synchronized re-initialization of all units, it is the only one to fire. In general, the 
winning unit is determined by a combination of initial network activation state and input 
magnitude. With appropriately chosen parameters, however the reset state establishes 
initial conditions which make the winning unit decision independent of initial state and 
dependent exclusively upon the X{ inputs. 

For the winning unit to exactly correspond to the one having the largest input, it is 
important that initial condition independence be maintained. To guarantee this indepen- 
dence in the PWTA network described by (2) parameters are chosen such that all units 
reset to an identical initial condition. 

All activations in the PWTA of (2) will reset to near-identical initial states if parameters 
are chosen such that v* < V th v' k - V tl and (3 < A [11]. During the output firing phase, 
these values cause the losing units to approach V t i well before the winning unit does. When 
the winning unit reaches V t i , it terminates the output firing phase and all activations cease 
to decay. Theoretically, the losing units only asymptotically converge to V t i while the 
winning unit converges via a truncated exponential. Even though this is mathematically 
imprecise, in a practical sense it can be assumed that /? is chosen large enough with respect 
to A for losing units to converge to within the limits of finite precision hardware well before 
the firing phase ends. 

A geometric interpretation of ideal PWTA network operation with constant inputs is 
illustrated in Figure 2. Each loop in the state diagram corresponds to one firing cycle. 
Unit activations are reset to V t i at state So in the diagram. The input integration phase (1 
and 3 in the figure) begins at S 0 and terminates when the V t h threshold is reached. That 
is followed by the output firing phase (2 and 4 in the figure) during which unit activations 
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Figure 2 : Ideal PWTA network operation 


decay. In the figure, trajectory 1-2 corresponds to the path followed when unit j wins, 
and trajectory 3-4 to when unit i wins. Which unit wins is determined by the one which 
first reaches V t h ■ That in turn is determined by the state trajectory during the input 
integration phase. With constant inputs, that trajectory is linear with slope proportional 
to the quotient of the input signal magnitudes. A “winning boundary” for constant inputs 
can be identified by unit slope (dashed line in the figure). This geometric interpretation 
of PWTA operation will prove useful in a later discussion of finite precision e ffe cts. 

Figure 3 illustrates the operation of a 2 -unit PWTA network in response to a smooth 
transition between two inputs. In this example, input signals xi and x 2 move from 0 to 1 
and from 1 to 0 respectively, crossing at t= 10 . The parameters chosen for this simulation 
are Vti = 1, V t h — 4, K = 0.1 a = 0.1, /? — 1 . 2 , 7 = 1.3, and A = 0.6. The activation state 
space diagram for this simulation is shown in Figure 4. It can be seen how the reset state 
with these parameters assures input order preservation. 


4 CMOS Circuit Implementation 

By way of introduction to the CMOS PWTA network, a CMOS implementation of a 
pulse firing processing element shall first be considered. Figure 5 shows the circuit for 
an impulse neural circuit as described previously in [ 8 ]. For simplicity, 7 = 6 , and a is 
for practical purposes nonexistent by virtue of the low leakage currents exhibited in MOS 
technology. C K includes not only the ideal capacitance of a poly -1 capacitor, but also 
stray wiring capacitance and the input capacitance exhibited by the Schmitt trigger G. 
The Schmitt trigger provides high voltage gain at the threshold voltages V t i and Vth, with 
positive feedback from the output establishing the active threshold. The Schmitt trigger 
output can be expressed in terms of the hysterisis function g of ( 2 ) as G(v) = Vdd9(v)- 
corresponds to the channel conductance of M2 which operates in the active region when 
an output pulse is generated. Further details regarding the operation of this circuit are 
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Figure 3: Inputs, state variables and output of a 2-unit PWTA network 



Figure 4: State trajectory of a 2-unit PWTA network simulation 
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Figure 5: CMOS implemantation of a pulse firing processing unit 
provided in [ 8 ] . 

A CMOS implementation of a PWTA cell is shown in Figure 6 . The basic elements of 
the CMOS impulse circuit are augmented by additional MOSFETs which establish various 
parameters associated with the ideal model. Variations of this circuit having reduced 
transistor counts are also possible [ 11 ]. The circuit of Figure 6 simply represents the most 
general CMOS PWTA implementation consistent with the Equation ( 3 ) definition. 

Two local signals and one global signal control circuit operation in accordance with 
current network state. The local signals G, and G, indicate that unit i is the winning unit 
when Gj = Vdd and G, = 0 V\ These signals select the local firing response. Both the 
true and complemented global lateral inhibition signal F and F are derived by a pseudo- 
NMOS NOR gate and a CMOS inverter consisting of transistors Mil through M14 in the 
diagram. These signals are distributed on two wires between all cells of the AVTA network. 
NOR pulldown transistors (Mil) are distributed across all cells while there need only exist 
a single pullup transistor (M 12 ) and single inverter (M13, M14). When any unit in the 
network initiates a pulse, F becomes active, causing all unrls to enter the output firing 
phase. 

Transistor Ml disconnects input current x { during the output firing phase, causing it 
to be shunted into the parallel capacitance of some input circuit (not shown — see [ 8 ] for 
further details). Also during the firing phase, transistors M 2 , M7, M 6 , and MlO conduct, 
allowing some combination of the currents Ij through f 4 to flow. 

The circuit branch consisting of M2 through M4 establishes a current which corresponds 
to the constant I x = 7. Similarly, branch M7 through M9 establishes a constant current 
corresponding to I 3 = £. Ignoring the nonlinear component of channel conductance, the 
branch consisting of transistor MlO establishes a current corresponding to J 4 = \v, and 
the M5, M 6 branch a current analogous to (/ 3 — A)v,-. As with the circuit of Figure 5 , a is 
assumed to be negligible. 

During the output phase, the signals G and overlineG control the Unit response. If 
the unit is a winner, then G = Vdd, G = OV, and branch currents J 3 and J 4 are allowed 
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to flow. This establishes a winning unit response where the unit activation asymptotically 
decays to £/A, but is truncated at V t i when the winning unit terminates the output firing 
phase. If the unit is a loser, then G = OV, G = V D d, and branch currents I x , I 2 , and 
I 3 are allowed to flow. This establishes a losing unit response, where the unit activation 
asymptotically approaches Vjj at a much faster rate than it would otherwise as the winning 
unit. 


5 Finite Precision Effects 

Thus far, the effects of finite parameter precision have been ignored. Intra-cell parameter 
variation will contribute to deviations from the ideal performance previously described. 
This section focuses upon the parametric variations which will have the greatest effect 
upon PWTA performance that also are the most likely to occur in contemporary CMOS 
fabrication processes. 

The overall function of a PWTA network is to select the input signal having the greatest 
magnitude. Inspection of Figure 2 reveals that there are two potential error sources which 
can interfere with that function. These are errors in the determination of the initial 
network state, So and deviations in the position of the winning boundary. These variations 
effectively give an unfair advantage to some processing units, sometimes allowing units to 
fire even thought their inputs are not necessarily the largest. Fortunately, it can be shown 
that this occurs only when two inputs have very nearly the same magnitude. Units having 
input signals which are “clearly” not the largest will remain quiescent. The definition of 
“clearly” is expressed as a hysterisis deadband which naturally occurs around the winning 
boundary. This hysterisis arises directly from parametric variation. 

For the remainder of this section only constant inputs will be considered. This allows 
for the analysis of parameter precision effects while the network is in a steady-state op- 
erating condition. Figure 2 illustrates network operation under ideal conditions when the 
critical parameters V t i } V t k, 7 and j3 are assumed identical across all units. Under these 
conditions, unit i wins if 


dvj 

with unit j winning otherwise. ^ — 1 corresponds to the ideal winning boundary 
(dashed line) of Figure 2. 

Variations in the scaling constant k yield an inaccurate winning boundary definition, 
k is determined in the CMOS implementation by the MOS capacitor C K in the previous 
circuit diagram. Variations in capacitor geometry will lead to inter-unit k variation and 
subsequently give those units having a smaller k an advantage in the race toward VJfc* 
Recognizing that 
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yields the decision rule that unit i wins if 


dV{ Kj 

d.Vj Ki 


( 16 ) 


This rule reduces to the ideal one of (14) when /c* = kj Although all units are initialized 
to the same state at So, the decision boundary is shifted such that one unit is favored over 
another. 

Variation of C K alone leads to finite precision for the winning boundary. Parameter 
variations which affect the initial network state and the unit firing thresholds lead to more 
complex hysterisis effects at the winning boundary. Firing thresholds, as defined by V th 
in the Schmitt trigger are typically determined by device geometries. The initial network 
state is determined in part by Vj; which is also dependent upon Schmitt device geometries. 
The other part of initial network state is determined by the geometries of M2-M6 and 
M10 of Figure 6 (corresponding to /?, 7 and A in the ideal equations). Not only do such 
variations change the slope of the winning boundary, but the slope is also dependent upon 
the winning unit as well. IJnder these conditions, the current winner is favored by the initial 
states such that a new winner must have a significantly larger input than the present one. 
These effects can be used to extend the decision rule expressed by (16) to one where unit 
i wins if 


dvj Kj Vthi - Vtn 

dv : k, V thj - V 0J v 11 > 

where Vo,- and Vo j are determined by the combined variations of M2-M6 and M10 
between units i and j . It can be easily verified that this decision rule reduces to the ideal 
case when there is no inter-unit variation. The effect this has on overall PWTA function 
is to introduce a hysterisis deadband which only affects close decisions. 

6 Conclusion 

An ideal linear model has been used to establish a general basis for PWTA function. 
This model improves an earlier one based upon presynaptic inhibition in two ways. The 
new model uses 0(n) interconnect for lateral inhibition and does not require an external 
reset signal since it is fully asynchronous. Furthermore, it provides information regarding 
how strong a winning input is — a feature not always found in winner- take-aU networks. 
Model parameters can be chosen to guarantee ideal winner-take-all function given precise 
parameter specifications. The model is also fully compatible with previously established 
asynchronous pulse firing analog neural ICs. 

CMOS PWTA circuits have also been presented. These circuits necessarily deviate 
from the ideal linear model, but they can be designed to exhibit similar behavior simply 
by accounting for the nonlinear characteristics of the electronic devices they employ. Non- 
ideal effects arising from practical implementation considerations have also been addressed. 
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Figure 6: A genral CMOS PWTA cell 


Parametric variation between pulse-firing processing units leads to a finite decision accu- 
racy and the potential existence of a hysterisis deadband. 
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